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' Illinois-funded Project IMPACT was designed to 

^dentify and de/elop procedures for complying with the impact 
^requirement of Public Law # &4-4B2, with regard to the effect of 
vocational^ education research and development on vocational education 
programs. The study addressed itself to three problem areas: (1) how 
to define ^impact, (2) how t*o assess impact/ and (3) how to show a 
cause^nd-ef feet relationship between project activities and changes 
in vocational education teachling-le.arning- situations. The project 
also inyfe^tigated how to predict the probability of impact, and hpw* 
i *o manage ongoing contracts to increase impact probability. In this 
first volume of the phase, 1 reports, Project IMPACT is defined, 
principles of assessing impact are determined, and method^ of study 
are decided through a review- of literature and an analysis of case 
study methods* The review of literature on planned change aad impact 
assessment showed that evidence of impact may be direct indirect; ' 
that the, time between caufee and effect of impact is longer for 
intellectual, products than for tangible products, andf'that it is 
desirable to assess impact of, a project at the school district level 
•rather than at higher administrative levels. .It also showed that 
numerous characteristics which are associated with high versus low 
impact have' been identified and that the degree to which thes£ 
characteristics^ ire present or absent may be useful in predicting 
actual impact. Through the analysis onf case study methods, procedures- 
for assessing impact of vocational education projects through case \ 
Studies were established. ^These procedures were followed in the case* 
studies in volumes 2-8 of^this series (see note)>. (KC) 
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Volume I is part of the final report for>hase 1 of the Illinois 
state -funded project "Development of Procedures for Assessing ,the 
Impact of Vocational Education Research and Development on" Vocational 
Education" - (PROJECT IMPACT). * ^ • 

This document contains an introduction „to PROJECT IMPACT, a 
review of literature and . views of experts in regard to., planned ' educa- 
tional change and 'impact,, the methods used to conduct impact case 
studies, a concise listing of references used during Phase 1 activities, 
and appended 'materials such as the initial project proposal and data 
collection instruments which were developed for us in the case studies/ 
Supplemental reports of the project are contained in the following vol- 



umes; 



1 



Volume 2 
» 

Volume 3' -7 

Volume 4 
Volume 5 
Volume 6 
Volume 7 -- 
Volume 8 

Volume 9 



J 



A Case Study of the "Illinois Occupational 
Curriculum Project" 

A Case Study of the' "Illinois Network of 
Exemplary Occupational Programs for Handi-^ 
capped and Disadvantaged Students" 



A Case Study of "Illinois Projects in Horti- 
culture" 

A o d Case Study of "Illinois Career Education 
Projects at the Awareness Leve/l" 

A Case 1 Study of the "Occupational Survival 
Skills Project" 

Case Studies of- '"Two Illinois School Districts 
With Innovative Vocational Education Programs" 

A Field Study of ."Predicting Impact of Fte- 
seafch and Development Projects in Vocational 
and Technical Education" \ 

Executive Summary of Volumes 1-8* and Con- 
clusions ar\d Recommendations 4 for Assessing 
the Impact* of yocatior\al Education Research 
and Development on Vocational Education 
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* . " Chapter 1 

• . ' i ' Introduction to PROJECT IMPACT 

Federal legislation of 1867 estabJished. the United Stages Department 

— ' * . 

* of Education plater USOE and' now Department of Education) and re- 
quired that it collect "sdch statistics and facts as shall show the condi- 
tion and progress of education in the several states and territories, and 
of diffusing information respecting the organization and management of 

'schools and school systems and methqcfs of teaching" (Suppes, 1978). 
In order to carry out these responsibilities, USOE has funded numerous 
research and' development (R, & D) activities in some areas of education 

<t; 

for over 100 years. Vocatic}j*al education, however, was not a U£OE 
responsibility until the passage of the Smith-Hughes Act in 1917. Very 
little research was done in this area from 1917 to •1963." 

Prior to 1963, research in vocational education was done primarily 
by professional associations and doctoral, students. Large-scale R&D 
activities were first made possible By the Vocational Education Act of 
1963 (Public Law 88-120) when Congress appropriated 10% of" the voca- 
tional education moneV for each fiscal year to g© toward R&D activi- 
ties. .By, .1965, Illinois and 23 other states had established * Research 
Coordinating Upits (RCUs), In 1968, Congress^evised the Vocational 
Education fyct of 1963 with' the "1968 Amendments" (Publr'c Law 90-576)- 
and made the' RCUs a permanent part of vocational education. By 1969, 
each state and territory of the .United' States had an RCU. 

In 1972^ USOE asked the National Academy of Sciences (NAS) tQ 
assess the impact of approximately 250 million dollars spent by USCTE .on . 
# K vocational education R&D activities during the' 10 years between 1965 
and 1974. The fJAS Committee on Vocational Education Research and 



■Development (COVERD, 1976) reported that the R&D of the decade 
studied did not have documented, widespread impact on the knowledge, 
•skills or empjoyability of \arge numbers of stud'ents except in curric- 
ulum development. National evaluations of vocational R & D in addition 
the COVERD report have been similarly critical (Development Asso- 
dates, 1975; Rand Corporation, 1975; Comptroller General of the United 
Stated, 1974). ''Acting upon these n^forts, Congress passed the Educa- 

tional Amendments . of h9?6 k (Public Law 94-482) which mandated that % 

/ 

contracts .for R&D in> vocational education would not be made unless 
the applicant could '''demonstrate a reasonable probability 11 that the 
contract would result in improved teaching techniques or curriculum 
materials that would be used iA a ."substantial number of classrooms or 
other learning situations within five years after • termination of such 
contracts 1 . 1 ( Federal Register , 1977). * ^ 

At the same "time that evaluators said they could not find very 
much evidenoe of impac{ (especially on students), they also said that 

/they ' lacked impact ' measures _ and impact data. Moore and Magisos 
(197?) reviewed the COVERD . report and other national evaluations of 
vocational education R &,D. They concluded that the criteria for 
subsequent^ evaluations "shpuld'be explicit enough to provide direction 
for* instrument design and data collection." They also concluded that 
"total evaluation ought to be programmatic, linking several projects 

^together, so .that the summative evaluation is broadly based/ 11 

I The purpose of PROJECT IMPACT . was to identify and develop 

i 

procedures for. complying with" the impact requirement of Public Law 
94-482. The study addressed itself to three problem areas: -,(1) how to 
define impact, (2) tiow to assess impact, and (3) how to show a cause-' 
and-effect relationshffObetween project activities .and changes in the 

••• 10 



vocational educat^oa teaching-learning situations. The project also ad- 
-dressed^w<r : subsidiai\ problems: (1 ) ,hovy to predict the probability of 
^^mipact and (2) how to manage on-going contracts to increase - impact* 

probability. ' ^ 

* 

The methods used to conduct, this study were to review literature 
related to the problem areas, to interview individuals who have ex- 

V perience and expertise in the problem areas, and to analyze the process 

> ** 

of impatt of -several programs of related projects which were funded -by 

. • * V 

the 'Illinois Office of Education's Division of Adult, Vocational and 

Technical Education (IOE/DAVTeT since it established an RCU and one. 

project funded by the Comprehensive ' Employment and Training Act 

^JfiEiA) — - . 

The major activity of ^TffT-stu<iv^v^ 
of related^ R & D projects. ' For the first year, it was decided to select 
"two. cases for "top-down" analysis and two cases for "bottom-up." analy- t 
sis (see Appendix A, p\ 43-)- The project staff referred to these types 
of retrospective analysis is "tracking." ft was anticipated that the two 
types of tracking would produce differenf insights about impact. For 
example, bottom-up tracking might better identify "bottlenecks" to im- 
'pact than would top-down tracking - while "top-down" tracking might be 
re effective in relating project intents and project outcomes. One 
staff member was assigned as "tracking manager" for each : of the four 
cases and, was instructed to keep -a detailed log of her/his; activity 
(e.g., identifying documents, retrieving documents, identifying key 
people and interviewing them, recording data, analyzing data). 
* Nominations .for the first four, case studies JWere sought from mem- 
bers of PROJECT IMPACT'S Advisory Committee, project consultants, 

i 
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and project staff members. The following cases were* sefected for 

/ , * , * ** 

"top-down" tracking: - x 

p 1 . "A/ Research and Development Project in Occupational Edu- 

% 

* cation" (The Illinois Occupational Curriculum - Project ~ 
I.O.C.P.) which was .developed by Joliet • Junior College and 

. ' funded by them and by IOE/bA VTE in fiscal years 1970-72. 

2. "The Illinois Network of Exemplary Occupational Programs for 

• Handicapped aYid Disadvantaged Students." At the time the 
case study was initiated, the network was in its fourth year 
of operatjLQ^^ IOE/ 

^^~-^"*~**^ DAVTE" had funded Illinois State University jto coordinate dis- 
semination for the Network's eight demonstration projects,. 
The third and fpurth cases, which were selected for "bottom-up" track- 
ing, were: ^ 

3. ^Illinois . Projects in -Horticulture. " IOE/DAVTE funded some 
4^ ' twelve R&D projetts in horticulture betweerv 1967 and 1978. 

Both private and public schools participated in these R&D 
efforts. 

4. "Illinois Career Education Projects^at the Awareness Level." 
IOE/DAVTE funded three or more major projects in this area 
between 1970 and 1978, and CETA began funding on# for the 
Illinois Department' o£ Correction in ' 1975. 

The remainder of Volume 1 is divided into , Chapters 2 and 3. 
Chapter 2 reviews literature pertinent to ( impact study as well as ideas 
which surfaced' during planning and review sessions at PROJECT 
IMPACT. • Chapter 3 contains a review of literature on case study 
methodology and the details of the tracking methods used in the first 
four case studies. j ' 
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Chapter 2 

- - Review of Literature apd Views of Experts on 

Planned Change and Impact Assessment 

Literature on planned change and assessment of programmatic 
impact, as well as views of experts which surfaced during the "brain- 
storming" sessions of Phase 1 of PROJECT IMPACT, are included' in the 
following review. The review is contained in two sections one on 
"planned .change" and one on "impact assessment." 

\ * ' V Prated Change N . 

Watzlawick, Weakland, amd..,Fis'ch (1974) described two types of 
change '-'first-order change" (where things have only been moved 

around) and "second-order change" (where a real difference has Been 

, i* * 

made). Methods of* effecting second-order change have been jnvesti- 

le- 
gated as' an area of study for about 20 years. Some of the most cited 

investigations of planned change were done by Lippitt, Watson, and 
Westley (1958); Bering *Benne/ and Chin (1961); 'Rogers (1962); 
Havelock, Huber, and Zimmerman (1<5'69); and ' Rogers and Shoemaker 
(1971 ). A national conference . on planned change was held at the 
University of Michig^ in 1973. In the published work of this con- 
ference (Havelock & Havelock, 1973), four major perspectives on the 
change process were presented as alternative models Jpr effecting 
Change: a Problem^S^lving model, a Social-Interaction model, a Re- 
search and Development model, and a Linkage model . Berman and 
McLaughlin (1974) pointed out that each of these models focQses on 
preadoption behavior (awareness, interest, evaluation, trial, adoption) 
and ignores the issue of implementation or institutional ^adaptation of an 

* ^ 

y 

13 • 
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innovative strategy. Berman and McLaughlin said that while these 
models may be adequate for technologies or technological products they 
are inadequate for educational innovations. They explained that a 

technology or a technological product is usually invariate "in its imple- 

9 Pig- 
mentation and in its outcome from one context to another, but an educa- 
tional innovation has • characteristics which usually make its installation^ 
variate . Characteristics of technologies or technological products versus 
characteristics of educational innovations were listed by Berman and 
McLaughlin as follows (pp. 9-10): 



Technologies or Technological Products 

- Sgecificity of treatment . 

- A clear relation between treatment 
and outcome; certainty 1 of outcome 

■ - Passive user involvement 

- A unitary^ adopter 

\ t *Clarity anc) specificity of goals 



Educational 'Innovations 

- Treatments are incom- 
. pletely specified 

- Outcomes are uncertain 

- Active user involvement 
*is required 

- The., adopter is* not uni- 
tary but a policy sys- 
tem or policy units 

- The relationship of pro- 
ject treatment to overall 
institutional goals is un- 
clear or unspecified 



Miles (1964)^described the installation of la' technology or a tech- 
nological product, as a mechanical process, and tt^e installation of an 
educational innovation as an evolutionary process. Berman and 
-McLaughlin called the evolutionary process a "mutation phenomenon" and 
explained that "innovation A may become innovation A when it is im- 

plemented in another .setting, and it may, be again changed to become 

3 * / ' 

A as it is v carried .out at yet another site." Trtey pointed out that the 

process which really occurs when an educational 'innovation is imple- 
mented is an organizational process. This is a .two-way process of 
adaptation in which the innovation . is modified to suit the Institution, 
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and ^the institution changes to some degree to accommodate the innova- y 
tion. Because of the mutation phenomenon, there'is very limited utility 
in pooling data on* different schools Or in looking only at relationships 
between treatment and student outcomes when one is trying to |ssess 
change. 

\ Pincus (1974b concluded that postadoption behavior of the school is 

. Nx V ' • - 

one of the variables affecting student outcomes, pincus suggested that 
project planners 'should consider the institutional change which" any 
adoption would require. Pincus categorized types of change -as follows: 
Change that increases the level of resource use only 

- . Change that affects the resource mix 

Change t,hat affects instructional processes dr methods without, 
altering resource levef mix 

ange that affects administrative management without signif- 
icant alteration of the organizational power structure 
• * * 

- - Change that affects either the organizational structure of the 

school or the school's relation to external ^rthority 

Se?cJman (1978) hypothesized that rates of returns on investments 
in educational innovations are related to the types of changes required 
for adoption (or adaptation). For example, he said, "Changes ih role 
relationships take more time than changes in technological processes or 
products." 

r 

A new model for planned educational change was introduced 1 by 
Berman and McLaughlin (1974). They proposed a three-stage process 
model which focuses on postadoption rather than on preadoption behav- 
ior. "First is a Support stage, second is an Implementationl^pge, ^nd 
third is an Incorporation stage. The Support stage involves a series of 
decisions on the part of potential users of an innovation. These deci- 
sions go far* beyond a rational consideration of the merits of an innova- 
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tion, and the decisions are essentially political. That is, costs and 
benefits are institutional or personal rather than budgetary. The criti- 
cal decision's of "the Implementation stage are made .by schQol districts.' 
The extent to which ah innovation is adopted (or adapted) depends on 
local interests, incentives; and priorities. The Incorporation, stage is 
the point at which an innovation loses its project status and becomes 
part of the school district. As "seed money" is withdrawn, cost/benefit 
questions become central to making decisions about incorporating an 
innovation in parX^r in whole. Although continuation of an innovation 
at district expense is a measure of high project impact, it cannot be 
said that discontinuation is an index of low project impact., Tl^s/ari- 
ables affecting Incorporation are many and complex. 

A considerable amount of the literature on planned educational 
* 

change focuses on adoption, implementation or incorporation of ,what 
Swanson (1976) called physical products as opposed to intellectual pro- 
duets of educational R &, D activities. Physical projects such as stan- 
dardized tests and equipment, texts/ films or other teaching materials' 
are easily recognized and understood. \^They can be packaged and 
transported. On the other hand, intellectual products such as survey 
data or new analytical tools may only be recognized and understood by 
the people who develop and use thepn . They are more difficult to pack- 
age and to trace. Battelle (1973*), traced 10 socio-economic innovations 
and found that basic research provides the origins from which science 
and technology can advance toward innovations. Only' 1 of^the 10 inno- 

vations studied by .Battelle was an intellectual rather than a physical 

' • f 

product or process. The intellectual innovation .(input-output economic 

.. * 

analysis) was first conceived in 1936 but was not accepted widely until 
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1964. • Although there were many related research projects, they, cul- 
minated iQ a significant impact only after 28 years. The time /for impact 
of the 10 innovations ranged from to 27 ye§r*s and averaged nineteen 
years. 

SeveraJ studies (Battelle, 1973; Berman & McLaughlin, 1974; Illi- 
nois Institute of Technology, 1968; Magisos & Moore, 1977) have ob- 
served that measurable »o? "observable 'changes (impacts) which result 
from R & t) activities are cumulative. Changes whigh are"" not statis- 
tically significant within five years, of implementation may be statistically 
significant within twenty years *and vice versa. Longitudinal studies of 
si^iph a nature are rarelf done. As more time goeC by, it becomes in- 
creasingly difficult to relate cause and effect because of an increase in 
the number of intervening variables. Tt has also been observed thaT 
the^best time to assess impact varies greatly. It would require tremen- 
dous amounts of money and manpower to monitor each of the impacts of 
a gro(jp of related projects for ten or twenty - years. Ev^ns (1§78) 
suggested that "instead of running a movie camera until we get what 
we're looking for* (a statistically significant impact), we -might analyze 
.types of physical and intellectual' products or processes and learn 
"when to have the camera there." In spitje of these problems, assess- 
ment of the impacts of educational R&D activities has been attempted 
rather frequently . during the seventies." Major efforts in this area are 
reviewed below. 

Impact^-Assessment v 
The review of literature and ideas on assessing impacts of educa- 
tional R&D activities is organized to help answer the following qu^s- 



tion: What is educational? impact? How can Ft.be recognized, hqw.can 
it be measured, how can it be predicted, and how can it be facilitated? 
What Is Impact and How Can It Be Recognized ? * * 

Dick (1976) defined impact as ' measurable phenomena of positive 
or negative value -- which result from the completion of a project or 
program. According to Dick, these phencfmeni (dependent variables)* 
will not occur in the absence of the project or program (independent 
variable). Miller and Miller (1974) concluded that impact has at least 
two broad parameters --/^ intended impact and actual impact. Their 
concept of impact is distinctly differed than Dick's, for it is clear that 
Dick considered impact. to^be actual impact. However; both studies 
described similar methods of 4 assessing impact and emphasized that 
evaluators need access to data on intended impact in order to find 
evidence of actual impact. 

i 

' Stake (1978) .and other educational evaluators at the Center for 
Inktructional (Research and Curriculum Evaluation (CIRCE) at the Uni- 
versity of Illinois concluded that' much significant impact in education 

t 

can be seen but not measured. Using case study methodology, CIRCE 

has recorded observable and measurable evidehce in what they Qa 1 1 

naturalistic (rather than formalistic)-evaluation. ' 

Documentation, of the impact which nine cases of basic research 
.4 « * 

have had on educational practice wa's recently published (Suppes, 

• * 

1978). All of the cases concerned intellectual rather than physical 
products. Ir\ assessing the impact of the theory of psychoanalysis on 
American elementary education, 'the researches stated \hat the ideal 
evidence .would- have been ,J data obtained from extensive observations of 

is 
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elementary school classrooms during the period 1900-1957. 1 .However, 
su^h data were,, not available and indirect evidence of impact was 
sought primarily published educational literature. , For example, 

textbooks used in teacher education programs were examined for the 

y 

number of^ pages giving coverage to psychoanalysis. The average 
number of pages for each decade was then calculated and graphed. 

Impact of .an educational project can thus be considered as the 
measurable or observable changes/influences (positive or negative) that 
Lare intended, immediate and subsequent. These changes/influences 
could be both quantitative as well as Qualitative and relate to the pro- 
cesses, products, inputs and environment of/the educational system to 
which the project is directed. ThecsfeCe; a chain of assessment at the 
planning, formative ( summative dnd subsequent stages of a project mus,t 
be carried out in order to assess a project's ultimate impact. 

An impact which is intended may or may not be realized within- a, 
givefi length of time. Impacts which do occur (actual impacts) may or 
may not have been intended.^ The groups impacted may also be intend- 
ed or unintended. Evidence of impact may be direct (such as in class- 
room ^bservauobs) or indirect (such as textbook citations), 



, ** Miller and *Mi)Jer (1974) reviewed over 700 federal and state R&D 
projects in vocational education and found that most' impact statements 
>yere of /'poor pua(ity." Most evidence cited in the impact statements* 
they quoted liad to do with numbers of reports written or\ disseminated , 

lumbers oft copies sold, numbers of inquiries requesting information, or 



• ' 1 4 ' ; 

The authors *stat§d that- this period covered the time from' the be- 
ginnings -of references to psychoanalysis in the American literature until 
the' death of the progressive education movement and the launching of 
. Sputnik.. f 

*< " ID . • •'/. 



numbers of students served. There was practically no evidence of 
measurable change. ^ 

Crawford, Kratochvil, and Wright (>972) analyzed 21 exemplary 
projects in education. £ach of the projects was profiled on 54 charac- 
teristics; Some of these characteristics had to do .with implementation 
and some*' to do with impact. Those which had to do yvith impact 
were as .follows: 

Percentage ,of target population ^ the" intended^ grade level' 
Number of field test sites 

- . Number of students in field tests 

Cost of product p$r student per year 

Number of -reported studies on effectiveness of product on 
affective gains,, cognitive gains, and facilitating gains 

Number of "schools in which product is used 

Number of students (in thousands) using product 

- ^Number, of states where product is currently used 

, - Amount of exacted use by a specified year # 

Only 1 of the 21 cases analyzed by Crawford et al' was a voca- 

tional education project, "The Cluster Concept." The only impact data 

reported for it was that its intended target group was*11th and 12th 

graders, but that only 40% of its actual target group consisted of 11th 

) - k 

and 12th graders. ^ 

♦ 

Ramp (1976) attempted to assess impact of 116 projects funcfbd by 
IOE/t)AVTE under, the 1968 Vocational Amendments and founjd that: 



f 93% of the projects were "being, utilized to some extent at the 
local, state or national level" . ^ - ; 

40,00(Tcopies .pf printed project materials had been distributed 
"to a wide range of educators" 

At least 7,500 persons had vPsited demonstration projects 
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More', than 11,000 schools had "In some . way been involved" 
with the 116 projects 

- ,. All local vocational directors were aware of at least 2 of x the 

116 projects . V| 1 

- , About 75% of project personnel reported that involvement with 

funded projects had been beneficial to their careers ' 

COVERD (1976) found that research developed. a cadre of research- 
ers: vocational education researchers became better researchers while 

social science researchers became knowledgeable of vocational education 
s* - * t * ■ 

I 

problems. However, few of the latter group continued vocational educa- 
tion researcff'after their initial* involvement ended. , 

Berman.and McLaughlin (1974) stressed that in order tolshow.^hat 
particular impact would not have occurred in the absence qf an R&D 
activity, evaluators must have data related to the conditions , of the 
^target group before and after the R&D Activity. Dick .(1976) went 
, further. He argued that little evidence of jmpact would be f<?und if 
only pre and post measures^ of summative nature were used. It was 

0 V 

Dick's thesis that the impact of summative*evaluation *is strengthened to 




'the extent £hat formative evaluation is employed. Dick advocateqL^on- 
tinuous evaluation of a project and suggested using two evaluators 
one internal and one external . The intergal evaluator 'cbll^pts .raw data 
and^s part of the project team. He/she should be inyolv^d as early as 
^^he^^toposal-wr^ting stage' of the project. The iexterrtal evaluator^ 
collects raw data but does examine it f« accuracy and .reports to 
the funding agency. Dick pointed out other (more important) functions 
of, the external evaluator. - There js a likelihood that the' external 

evaluator will identify problems which have gone undetected by project 

**■ % 

personnel. -Through discussions with a variety of staft and"* users, 
he/^he can gain a unique perspective on the project and provide* insight 
into difficulties. Often, the external evaluator can informally 



™ — ' — i ■' 

20 



supgest alternative solutions which might not otherwise have been 

• 'considered. : * . 

. . * Y • - ' " • 

Dick emphasized the importance of ^valuators having access to 

process -data, but Miller and. Miller (19t4) t went a ( step further. They 

emphasized the need lor planning , data' and concluded that "clear cri- 

„" terion measures" should be established before funding/; The developed 

r * - 

guidelines for specifying impacts by suggesting^ that project proposals 

I 

and reports should routinely include concise listi?iqs of: 9 

I 



1 the objectives of the project and a measure of how well those 
objectives will be (or were) reached. 

• 2. The numbers of schools, staffs, and students who will be (or 

were) involved and the proportion those are (or were) of the 
j total .in the district or state. 

* — - m 

r m 

3. The previous research which will be (or /was) specifically 
used as a resource for the project. 

■ ■'■ ' ■ ? • 

Miller and Miller cbncluded that: r * y / 

* .i « * ~ 

Impact — actual measurement of change — should fee a 
required . part of all projects, with teeth built in -and ( 

• funding provided. T^Tat is, if the impossibility of mea- 
suring, impact is nqr agreed upon .at the outset of the 
project, those who proposed the project should' be hjsfci 
accountably/ for impact. They should be" penalized when 
they do, not take the time ttt*-measure it,' or when they 
were wrong in sayjfig -the i/fpact could be- measured (p. 

.59). > • ~* K . ( ; 

De Neufille and 'Stafford (1971) developed-* an impact-incidence 

^matrix which was included in Systems Topis for ' Project . Planning (fDeip 

et al., 1977). The matrix provided cells for recording observable 

(qualitative)? as well as measurable (quantitative) data. Also included 

m x the matrix were cells for indicating target groups .to be impacted 

(see Fig. 1). . - 

How cap impact be measured ? - - • - ^ 

Actual irppact may be assessed art any or ail of the foFlowing levels 
\ ' . * . ' 

where educational fmpact occurs (Swanson , *197§^ „ ^ - 

On 



Groups impacted 



impacts 



Directly 
estimated 

' $/ , 

a 1 a 2 a 3' 



Indirectly 
estimated 
$ 

b 1 b 2 - 



Estimated 
numerically 
not in $• 

c 1 c 2 . . : 



Estimated 
qualitatively 
in words 

<V d 2 ••• • 



Directly 



Indirectly 



B, 
B, 



Special 
interests 



J Figure 1.** Impact-lnciderrce l\^tpjx for Cost Benefit Analysis * 

• 23 * 

* Systems Analysis for Engineers .and Managers by Richard DeNeufville and Joseph H • 
Stafford. Copyright -(c) 1971 by McGraw-Hill Book Company. 
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National level (e.g., Congress - educational legislation, # 
USOE - educational funding and regulation) 
• - State level (e.g., IOE, Assistant ^Superintendent for DAVTE) 
Regional level (e.g., Director of Area Vocational Center or 

Technical Institute or President of a Community College) 
Local level (Superintendent of .Schools for a Local Education 

Agency - LEA) 
/Unit level (Principal of ah LEA school) 

- Department lev^l (Chairperson) . \ ' 

- " Classroom level (teachers, students) 

-Community (parents of students, press, public groups) 

Some tangible products of impacts which may be measured and/or 
observed -at these various Jevels are changes in^ 
Capacity • 

Actual enrollment * - — « ___number and percentage, 

J - Attrition 0 f client system. 

- • Organization 

Attitudes (j&.g., acceptance, morale,\^elf-esteem) 
Efficiency '(e.g., . time required for n^n-instructiori'al paper-' 
9 - 'work, time, required to C9mplete training) 

Attendance (e.g.,|rless tardiness, fewer absences, Jess tru- 

^ncy) < 
•Status (of minorities) 
Achievement r f— 
Kpowledge _ 
Skills ' 
* Costs 



N 



y services^ ,. ^.national 

^\ according,.^^ 



Numbers of requests'for^ ~ , 

\ , ^ ^ to eve ; \ 
* ; „ N products"^ ^cl^ssroom 

Unemployment/underemployment 



Poverty/wages . 
-Crime, vidlence, vandalism 



5> Researchers at' The National Center for Vocational Education Re- 
search (1979) developed a chart ^which shows types of educational 
impacts on various groups that might be observed or measured (see Fig, 
Z). - * 

O'Boyle (1974) - pointed out the need for identifying unique criteria 
for certain types of projects. For example, when evaluating manpower 

-a 

training projects,, criteria such as "economically independent" or "em- 
ployed in the/ area for which tr^ned" are not as relevant as "having 
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4 Group Impacted ' f 




TYPE OF IMPACT 


SE/V/LEA 


SCHOOU 


.STAFF STUDENT ^ 


LOCAL COMMUNITY 


Attitude/Pohcy 


Changes in attitude 
toward females in 
administrative roles 
in vot 6u 


Changed in 

enrolment 

policies 


Changes in 
attitude to- 
ward voc ed 


Changes in commun- 
ity attitude toward 
voc ed 


.Knowledge/Skill 


Changes in evalua- 
tion knowledge and 
skills 




Changes in Changes in 

teachers students' 

occupational 

competencies 9 
<» ' * 


Changes in parents' 
knowledge of voc ed 
job options for their 
children 


Behavior/Practice 


Changes in 
state plans 


Discard of 

existing 

practices 


Changes in - Greater enroll- 
teaching 4 ments/place- 
practices ments of females 
in non-trained 
occupational pro- 
gram/Jobs 


Increased voting for 
school levies rejate^i 
to voc ed 

* • 


* 


c 

Figure 2. Potential 


Indicators of Change* 






* The National Center "for Research in Vocational Education. 
; * Columbus, OH: The Ohio State University, 1979.; 
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earnings which represent at least 60% of a poverty-level income" or 
"unemployed less than 10 weeks of the year." O'Boyle said that some 
of the pretraining data whidh would be necessary for assessing such 
outcomes are not usually collected and that this problem must be reme- 
died before impact of future manpower programs can be assessed. 

Henning (1975) pointed out the effects of research design on 
impact assessmenf. He saig that quasi-experiment'al designs have more 
impact on the present correctional system than dp. experimental designs. 
Cheney-Stern (1977) used a quasi-experimental design'"to assess the 
impact of career education on career maturity of male convicts in an 
Illinois State prison. She found that prisoners who completed a career 
education course had about twice the gain in career maturity that a 
comparable non -treatment group had over the same period of time. 

The National Council on Employment Policy (1976) pointed out the 
need for assessing the duration of impact. They concluded that high 
initial costs of a project or program may be justified by impacts (gains) 
which remain constant over time. Malhotra (1978) suggested that as- 
sessment of actual impact be divided into "immediate" and "subsequent." 
For example, a responsibility of each funded project would be to specify 
its intended outcomes (impact) and account for its immediate outcomes. 
However, .a State's funding agency would assess sustained and/or sub- 
sequent impact of a project or a group of related projects. Chen'ey- 
Stern and Evans (1979) concluded that this type cPlongitudinal impact 
assessment would require continuity in impact specifications for afl 
projects funded by the same agenpy. 

How can impact be predicted ? > * . ^ 

Several studies have identified characteristics associated with high 
versus .low impact projects (Battelle, 1973; Illinois Institute of Technol-* 

" 27 
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ogy, 1968; Moore & Magisos, 1977; McCaslin, 1978;. Murphy, 1974). 
The PROJECT IMPACT staff reviewed and synthesized these character- 
istics and found that many were extrinsic to the idea of a project. For* 
example, motivational factors, 'management factors, or even accidental 
factors intruded. From this list, Malhotra (1978) developed a "Rating 
Form of Success Factors for Predicting Impact of an Educational R&D 
Project" (see Appendix B). 

How can impact be facilitated ? 

t x 

Studies by Battelle-Columbus Laboratories (1973), the Illinois 
Institute of Technology Research Institute (1968), Miller and Miller 
(1974), and Suppes (1978) have all pointed to the need for research 
programs which form a coherent whole. There should be continuity 
between basic, applied, and developmental research. Miller and Miller 
studied over 700 federal and, state R&D projecrs^jn vocational educa- 
tion and concluded that problems were inadequately investigated before 
(assumed) solutions \o problems were designed and demonstrated. 
Cheney-Stern and Evags (1979) discussed the effects of "continuity" on 
impact. For, example, impact is likely to be facilitated if there is con- 
tinuity of effort on the part of researchers, continuity of goals and 
funding on the part of organizations which fund research, and continu- 
ity of reporting successful research results to consumers. , "Brainstorm- 
ing" sessions which were held during the early months of PROJECT 
IMPACT identified the following list of impact factilitators: 

The quality and quantity of the final report 
0 - The method of dissemination 
The timing of dissemination 
The target groups for dissemination 
The length of the project 

The risk of the project (e.g., |pw risk projects are usually 
low impact projects)' 
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The availability of funding * , 

The cost of the project v * 

The personalities of project personnel and disseminators 

* The reward system (reinforcers or contingencies associated 

with adoption) -3* • 

Copa (1978) suggested that impact might be facilitated if research 
investment patterns matched the service delivery systems. He pointed 
out that for the past 10 years, the investments in vocational education 
R&D activities have been in across-the-board (handicapped, dis- 
advantaged, etc. ) rather tVfen subject-matter specialties (agriculture, 
business, health, home economics, etc.) which actually deliver vocational 
education. 

\ Use of the "portfolio theory" 'to facilitate impact of an entire R&D 

program ra{her v than a project-by-project approach was suggested by 

PROJECT IMPACT .staff members. For example, a mix of projects which 

were low or high in* probable costs, risks and pay-offs "might produce 

greater impact that) a portfolio of projects, all of which were low in risk 

and high in cost and pay-off. The staff developed a matrix for classi- 

/ 

fying projects according to this scheme (see Fig. 3). 

Summary r Jt 

Vocational education R&D activities which result in changes in 

vocational education are said to have had "impact." Evidence of impact 

may be direct or indirect. The time between cause and effect of impact 

is longer for Intellectual products than for tangible products. Because 

of a mutation phenomenon which occurs when R&D projects - are 

adapted, it is desirable to assess impact of a project (or a group of 

/ 

related projects) at the school district (LEA), level rather than at re> 



gional, state or national levels. 
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Numerous characteristics which are associated with high versus low 
impact have been identified by researchers in vocational* education and 
other fields. The degree to which these characteristics are present or 

•SS5P * 

absent may be useful in predicting actual impact. & ' 

One of the greatest facilitators of R & D impact is a program whicfi 

has a coherent whole. Basic and applied research need to be funded 

adequately and carried out before developmental products, or processes 

V 

are attempted. * 

Jn order to avoid an overQoncentration on low risk, low pay-off, 

\ 

high cost projects, assessment of impact of research projects should be\ 
aggregated at the program level, rather than judged solely project- 
by-project. Assessment of actuaj impact of a project 1 \or group of 
related projects) is dependent on availability of planning and process 
data such as specifications of intended outcomes (impact). If cumula- 
tive impact (over timfe or geographic area) is to be assessed, it will be 
facilitated by the use of a standardized format for specifying intended 
and actual impacfs for all projects funded by the state agency. 



\ 
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Chapter 3 • * • 

Review of Literature on Case Study Methodology and 
— " Details of PROJECT IMPACT'S Methods 



A case study is "the'examination of an instance in action" (Walker, 
1974). The purpose of a case study is to enable the researcher to 
make generalizations from the instance studied to the class it purports 
to represent, ' to make generalizations from features of the instance to a 
multiplicity of classes, or to make generalizations about only the in- 
stance studied (Adelman, Jenkins & Kemmis, 1975). A variety of tech- 
niques (most of which have been developed by sociologists) is employed 
in^the case study method. 

Steps which are usually included in case study procedure are as 
follows: * 

Defining the^problenrv . y 

Reviewing research related to the probfem 
Developing objectives for the study 
Selecting the field setting(s) 

- ^ Making initial field contacts 

- ' • Gathering quantitative and qualitatiye data (e.g.,^ data on the 

. setting and participants, data from documents, data from 
interviews and questionnaires) 
Coding and analyzing Materials 
Reporting findings 
■ - Developing and validating propositions ' 

Drawing conclusions frorin the study ' 

Case study methods allow a wide variety of reporting forms such 

as coliage, film documentary, mixed-media presentations, role-play 

simulations, oral* feedback, quasi-jburnalistic reports, as* well as the 

more usual written reports (Adelman et al., >975)-y Stake .(1978) dis- 

of a singje^ca/e) 

from tHe multiple case study' project (a collection\of individual case 
. ■ ) 

studies). He -stated that special research procedures are needed for 
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the multiple case study project, but that, unfortunately, they are not 
well developed. Stake also pointed out that multiple case stuby projects 

4 ' 

"demand a form of linkage a manner in which to discuss their differ- 

9 

enqe.s'and similarities, 11 but that the topic is little examined in the 
methodological literature. * 

The case study literature does describe a * validafti^n technique 
called "triangulatioh" (Center for instructional Research and Evajuation, 
1978). The idea comes from sociology and navigation and it has to do 
with arriving "at the same meaning by at least three independent sourc- 
es." A controversial finding based on several accounts has more credi- 
bihty than .if it had been based on only one account. Therefore/ 
corroboration |is usually desirable. 

Ethical practices which 'ar*e common *in case studies are anonymity 
of sites and persons and confidenttality of information. These practices 
provide legal protection to case study researchers as well as fevacy to 
individuals and institutions. Therefore, towns, schools and people are„ 
usually given fictitious names in case studies. Clearance procedures 
are oftert required at the time fifiW contacts are made^ For example, 
any research' activity within a -school district may have to be approved 
by the Superintendent of Schools before any interviews with school 
personnel may be scheduled or before any documents may be released. 

Crawford et al. (1972) at the 'American Institutes of Research 
(AIR) developed criteria. for ca^e selection, from a variety of educational 
R 4 D activities. One hundred and seventeen exemplary products or 
processes wh-ich meX the criteria were identified/ and twenty-one/ were 
selected for an impact study. Six procedural steps were developed for 
the multiple case study project and are summarized below: 



Collect data on the origin of the idea for the product (or 
process). 

Collect data on the design and conceptualization of the pro- 
duct (or process*). 

« 

Collect data on the history of the development of the idea into 
a product (or process), including the source of support and 
< the identity of the devejoprers. *t\ 

Collect data on pilot and field testings of the product (or 
process), and the results of such analyses. 

Collect data on the, efforts- in dissemination of information 
about the product (or process). 

Collect data which might serve asTdirect or indirect measures 

of the impact of the product (or prbcess) in its operational 

form (e.g., the number of students involved, the number of 

items sold, surveys of student and/tfr teacher reactions, and 
summaries of evaluative studies). * 

It was suggested by Crawford and his staff at AIR that\iata be 

collected by (1) identifying, retrieving and reviewing 'documents (e.g 

Jetters, proposals, budgets, progress* reports, finaf 'reports, brochures, 

and lists of workshop participants) and (2) interviewing key people in 

^ • r 

person and/or by telephone and taperecordirja each interview^ In 

considering the best approach to reporting and collected data, Crawford 

et al. decided that each developed case study wouJd include. fo^r conrt^ 

ponents: o * 

A systematic narrative history based on a master outline, 
which included sections f6r a general description of the pro- 
duct (or diffusion, its adoption, and its future.) 

A I data record consisting of a systematic cioded matrix of 
quantifiable aspects of* products (or processes). Information 
in the matrix was "arranged and coded for possible key- 
punching and machine treatment of the data."* 

» 

A description of critfcal decisions in the history of the pro- 
* duct (or process). These descriptions included the decisions 

that had to be made,, the alternates available for each 
decision, the alternative selected,- the forces leading to the 
selection of a particular decision, and the consequences 
resdltirtg from the selected alternative. 




A chart or diagram which provided an averviet^of -the major 
.events in the 'history of the prpduct (<3r process). 

» ' . " S>\ ' 

Crawford and his staff, generally completed a rough draft of the 

narrative outline before site visits were made. Analysis df documents 

^provided the information #or these drafts. The interviews wer4 used to 

verify the information and to fill in gaps in the outline of the draft. 

» * 

McCaslin, Adams, and' Gross 01976) studied characteristics of 
higher versfls lower impact projects conducted by Kentucky's Research 
and* Development Upit from 1967 to 1975. In their multiple case- study 
project they developed y\d used three forms for collecting data one 
for describing the R&D project", one for ^ interviewing the director 
and/or the principal investigator of the project, and one for obtaining 
information" from' project participants. . $ 

Miller and Miller' (1974) developed a, series of matrrces~for % analyz- 
ing impacts. They compared intended versus actual impacts, funding: 
llevels, and* t^pes of problems for over-7Qp federal, and state R&D 
projects in vocational education. * V 

0 \ **'.V ' J- 

f Delp,\ Thesen, Motrwalla/ and Seshadn (1977-) cited'an "impact- , 
incidence matrix" whichf was developed by De Neufille and Stafford in 
1971. The matrix, which provided "cells for recording both quantitative 
and qualitative data as wfell as o information on the groups targeted *for 
impact r was describe^ earlier in this report -(see* Fig. 1, p. 21). * 

Although there .is not a well -defined methodology for conducting 
multiple .case study projects in general, <there are" some precedents in 
the field of education y^especially vocational education. Aftep^denti.fy- 
irtg and evaluating the case study procedures and tools which w er^ 
^variable, the PROJECT l#IPAC"f staff decided to adopt or adapt" .some of 
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them for gathering and recording data on their four case studies. The 

r* 

adoptions and adaptations were as, follows: 

- « 

Establish case criteria and select c3se§ from the nominations 
of 'Advisory Committee members, and consultants. (Criteria 
for case selection ^were established , in the project proposal 
which is contained in Appendix A „and 'the "Advisory Com- 
mittee Questionnaire" for case - nomination is contained in 
Appendix C. ) 

Collect data according .to the six procedural steps described 
by Crawford and his staff at AiR for "top-idowh" tracking of 
cases but reverse the- steeps for "bottom-up" tracking. 

Prepare a draft of a narrative record for each case by using- 
an adaptation of AIR's "Master Outline" (see Appendix' D). 

Conduct interviews using the narrative draft and/or the 
° ( Kentucky forms for project directors and project -participants 

as guides. (Copies of the "Project Director Interview Sched- 
ule" and o the "Participant Questionnaire" are contained in 
Appendices E and F respectively). * 

Arrange and code quantitative and qualitative impact data on 
an adapted form of the 1 matrix developed by De Neufulte and 
Stafford (see Fig. 4). ' * . 



Sta 



Describe^ the critical decisions in) the history of the case as 
suggested ' by the AIR researchers and corroborate data %frfcfv 
two or more interviews. r • 

Chart or diagram an overview of the major events in the 
. "history of the'case. 

v " • ' ' 

In addition to using these adopted and adapted methods\and tools, 

each ^tracking manager kept a detailed lo c g of events regarding his/her' 

case study. The priricipal investigator also kept a Ic/g on the activities 

of the over-all project. 

The results of case nomination, selection, and reporting proce- 
dures are presented below. 

The; questionnaire which was* developed, for nbminating cases for t 
this study was mailed to each memt^er 0 f the 'Advisory Committee and 
was circulated 'among the project consultants and .staff . ,Most responses 
to the 'qifestionnaire were received^ in person or by phpns rather than 
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Impact Specification Matrix 
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in writing. Most respondents nominated cases in only one or two cate- 
gories (rather than all five). M A Research and Development Project in 
Occupational Education 11 (The Illinois Occupational Curriculum Project — 
IOCP) and "The Illinois Network of Exemplary Occupational Education 
Programs f5r Handicapped and Disadvantaged Students' 41 were most 
frequently nominated and wegp selected by project staff for top-down 
tracking. Nomination and selection of cases for bottom-up tracking 
proved to be more difficult. The staff and consultants decided that a 
^roup of related projects should be identified for each of these cases. 
Because "IOCP" was targeted primarily on administrators, and "The 
Network 11 was x targeted primarily on teachers, -it was decided that a 
criterion for selecting the twfc "bottom-up" cases should be that their 
primary targets be students or both teachers and students. Another 
suggested criterion was that one of these cases should be closely re- 
lated to a subject-matter specialty as opposed to an "across-the-board" 
area. After considerable deliberation, the staff selected "Projects in 
Horticulture" and "Career Education, Projects at the Awareness LeveL" 
It was noted that^the horticulture projects were targeted primarily on 
teachers l^d students at secondary or post-secondary school levels 
while the career education 'projects being studied were targeted pri- 
marily on teachers and students at the elementary school level. 
* 

A tracking manager was assigned to each case to identify, retrieve 
and analyze documents pertinent to her/his case. Manager of t^-down 
case studies anajyzed project proposals and reports before they inter- 
viewed staff members or reviewed products of the projects in their 
cases. Managers of bottom-up case studies familiarized themselves with 

products of projects .before interviewing people who were likely to have 

I 

jf 
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adopted the pr&ducts. By having done so, they were able to identify 

curriculum materials or methods (or other products) which werg being 

used by te^^ers without having to ask teachers about particular^Rro- 

)jects or products by-name. * 

The following' six steps were taken to continue each case study, 

but were done in reverse order to tracking fro^ the bottom up: 

Data on the origin of the idea for the product&6ar process) 
wer.e collected. " i# 

Data on the design and conceptualization of £he product (or 
process) were collected. 

Data on the history of the development of the idea into a 
product (or process) were collected. 

Data on pilot tests, field tests and other formative evaluations 
were collected. 

Data on dissemination efforts were collected. - * 

- Data on adoption and summative evaluations (especially impact 
measures) were collected. 

After completing the above steps for her/his case, each manager 
investigated the future of the case, drew conclusions about the case, 
and made recommendations regarding further impact* investigations of 
the case. 
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) Appendix A 
The Initial Project Proposal for 
"Development of Procedures . for Assessing 

Vocational Education Research and Development on 
VocatiorVaT^ducation Programs 11 



the Impact oyf 



Research programs are expensive. They require . the. use of re- 
sources which are needed el<p*ere, and they can be justified, in the 
long run, 'only if they increase knowledge. Programs of. vocationa^ 
education .research and ' development, because of their relationship to 



vocational^ education', require a further justification: they must be 
judged in^terms of extent to which they have improved the quality of 
vocational education' service. In other words, they mutt be judged to 
havehad impact upon (and to have changed the behavior of) those who 
deliver vocational' education services, If the program has no impact. on 
vocational education, and 'ypcationali educators, it does not warrant 
continued support from vocational education funds. 

A research ancT development program \s composed largely, of pro- 
jects. These relatively .independent* ""and self-contained research or 
development activities can be ' characterized in a variety of ways: 
low-high risk,' low-high probable pdyoff, basic-applied, descriptive- 
experimental, etc/ Obviously, the probability of .impact from a "high-- 
risk, high payoff, project- is relatively fow, but if it.does pay off, it will 
have far-reaching -'effects; \f one knew in' advance what would be 
accomplished by a project, it would hot be necessary, in many cases, 
for the project to be undertaken at all'. Unfortunately, some evaluators 
•of resea'rch attempt to apply the "impact" test to each project, when it 
properly should be* applied only to a program. For example, the Con- 
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gress (in Public Law 94-482, iTitle II, Section 202) amended Section 103 
of the Vocational Education Act of 1963 to require eacj) research con- 
tract to include demonstration of reasonable probability of- " . . im- 
proved teaching techniques or curriculum materials that will be used ifi 
a substantial number of classrooms or learning situations within five 
years /. In trther words, each research. and develppment contract 
must have a probability of impact on vocational education teaching a and^ 



atijpe 

t 



learning. 

- Whether or not one agrees with this Congressional mandate, it is, 

in fact, the law. Managers of research programs, therefore, are faced 

with the necessity of assessing the probability that impact will occur, 

» 

and the moral obligation of measuring the actual impact of at leasj: a 
sample of completed research and development projects. This latter 
should be done, in any case, in order to assess the 'impact <pf the re- 
search and development program . 

As is often the case/ these necessities and obligations are easier to 



state than to do. Three major problems exist: how to define impact, 
how to assess it, and how to show a cause and effect relationship be- 
tween project activities and changes in the vocational education teach- 
ing-learning situations. Subsidiary problems include: how to predict 
probability of impact, and how to manage on-going contracts to increase 
the probability of impact. This project is concerned with the three 
m^jor problems, but it is believed that some suggestions related to the 
two subsidiary problems will emerge. 

Procedure 

1. Review literature -on impact of research and development and 
its measurement. f 
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2. Interview key personnel to identify ways now used to assess 
effectivenes|fcjtfncluding impact) of educational research and development 
programs and projects in Illinois. f TKey will also be asked atout effec- 
tiveness of current .methods^ aljout other methods which might be used, 
and reasons, why some of these methods have not been used. Persons 
to be, interviewed will include: administrators and managers of educa- 
tional R&D programs in IOE, in universities, in community cplleges 

and in K-12 school dstricts; evaluators of educational programs and 

V * 

projects in Illinois. ^ 

3. A series of one-day brainstorming sessions will be conducted 
to get ideas of hoyv to attack the three major problems outlined above. 
Groups will be small (3-5 persons per group), and will include repre- 
sentation from sociology, economics, philosophy, statistics, advertising, 
health, education, psychology, business management. Persons from the 
above disciplines who have worked on the following topics will f be invit- 
ed: • causatiogf, effectiveness,^, dissemination, adoption, institutional 
change, persenal changeftinnovation, research management and organi- 

. , zation, hypothesis formation and testing, readership, sales volume, 
assessment, evaluation, satisfaction, confidence tWing, accountability. 
Persons to be invited will be identified (a) by the advisory committee, 
(b) through the literature search, and CO through nomination by the 
persons* identified in steps 4 (a>) and 4 (b). 

4. Two major types of research Rroject tracking will be done, , 
one from 'the bottom up, and one from th^ top-down: 

a)' -Pour major refcerrt* changes in vocational education teach- 
ing and learning in Illinois at the LEA level will beVidentified . One of 
these will be related to curriculum materials production, e.g., IACP 
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Industrial Arts or nutrition education. The second will be related to 
vocational education ot a new target grpup, e.g./ the handicapped or 
incarcerated offenders.?. The third will be related to new cpncepts csT 
organization for 'vocational education/ e.g., area vocational schools or' 
CETA-YEJj&pA,'. The fourth will be related to r changes in* school goals, 
e.g . , career education or school-operated placements services. The 
project advisory committee recommend specific programs in each 

category and may recommend changes in categories For aach type of 
change, the determinants of change will b£ identified through, interviews 
with major change agents nominated txy currer/t actors, so that each 
actor will identify the person, event, material, or other occurrence 
which affected her/his actions. ' \r\ this way a chain^of occurrences* will 
be identified. If research results* are, a part of the chain, this ..w$ be 
noted, and its relative significance in the chain will be, assessed. 

b1 ) ^Vocational education research- and' development adminis- 
trators in Illinois will be asked to nominate the projecjt(s) with which 
they were mvolv^d which had the most "impact ^on vocational education 
teaching arid learning. Administrators bf research programs in IOE and 
universities will^be included, ^f^maximum of four projects will be se- 
lected, and by means of interviews with the principal actors, a chain of 
effects will be identified. In each case the 'actor will be asked who and 
what affected her/his actions, arid who or what was affected by the 
Victor. The point" at which knowledge of the research baseTwas lost will 
be noted. 

b2) Nominations will be sought (from this same group of 
administrators) of projects and programs which had strong p&tential for 
impict, but the predicted impactidid not occur. A chain of actions will 
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be constructed, and the point dt which .the .chain Was. broken will be 
nated and an assessment made of reas'ons for the break. 

5. Case studies of projects and programs will be prepared iden- * 
tifying: ■ *J , . 

a. measure^ of impact'^ 

v b. factors f actjli tating and hampering" impact 

c. factors facilitating and hampering recognition of impact 

■ ' ,d. factors facilitating 'and hampering linkages from one letfel 
\ to another . 

6. Tentative recommendations will be prepared for^ research*" 

administrators suggesting means, oj: * 

>- 

a. assessing potential impact 
- b,. assessing actual, impact 
. c. facilitating, impact ^ 
d( "facilitating recognition *of [mpact 

7. ^ copferenp§ of ^administrators, producers and^ consumers of 
vocational education research and development will be convened. > They 

will be asked to review c the case studies and tentative recommendations 

» * * » 

and to suggest modifications.. m . 

•8. Case studies and reqojrrmendations-will be revi'sed and drssem- 

inated as: & 

a. a monograph with executive summary 

b. a journal article/iuitable'forj publication in the Journal of 
Vocational Education Research * 

c. a. proposed or.al> presentation to the RCU . Directors and 
the American Vocational Assdciation 
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Project Professional Staff 

' u f f > 

Project Director, Or. Rupert N. Evans Professor, Department of 

Vocational and Technical Education/' University of Illinois, Urbana- 

Champaign. ^ 

'Principal Investigatory Dr.. Marilyn Cheney-Stern, Visiting Assis- 
tant Professor of Vocational and Technical Education. 
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Appendik B 

Rating Form of Success Factors for Predicting 
Impact of an Educational R&D Project 

A. Motivational Influences 0 1 

1. Motivation for the timely, acquisition of the 
fundamental knowledge useful for application 
in the v project. 

2. Motivation for* the timely -improvement of an 
existing product or process. 

3., Motivation fpr solving the problem or meet- ' 

ing the needs. \* 

♦ 4. Estimates of the potential market for the 
project/innovation. t ' 

B. Planning 

1. Provision for ohgoing and en£ evaluation of 
project effectiveness on- the basis of objec- 
tive data. 

2. Involvement of participants from the differ- 
ent client organizations more fully in»project 
planning. ^ 

3. Accessibility of planning- documents such as 
PERT or time effort charts and objectives. 

4. Development ao«r demonstration activities 
' supported t research activities in the 

plan. 

5. Planning for- more final evaluations bf the 
project, 

6. Fixation of priorities for attainment of ob- - \ 
jectives. ■ 

Provision for dissemination and utilization of 
project results. _ 

Provision for staff development of the client, 
organizations. 

High relationship between objective^ and 
-performance activities plantied. 
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10. Development of materials for use at the 
place of cMent organizations. 

11 . Consideration for manpower requirements 
for implementation. 

12. Provisions, for clear practical, gufdelines, 
delegation of authority and making available 
support to "the client organizations by the 
parent organisation. i 

13. Project planning based on need assessment, 
and priorities of these needs. 

14. Criterion standards established for evalua- 
^ tion of the project. 

15. Establishment of information system in the 
planning. 

✓ 

16. Participation of project staff in stating pro- 
ject goals and their perceived importance. 

17. Suggestion of goals and approaches by per- 
sons outside the project team. 

Management 

1. Earlier activities that established the prac- 
ticability of further development or the 
utility of further research. 

2. Provision for adequate* orientation for par- 
ticipants concerning their* role in the pro- 
ject. ^ * 

^3. Provision for adequate facilities for imple- w 
meriting the project effectively. 

4.. Coherent research and development strategy 
.with central unifying thrust. * 

Staff m 

1. Provision of staff for identification of scien- 
tific or technical information of relevance to 
""^■fctie • interests .and 'activities of researchers. 

2^ Project sponsored by organzation having 
prior experience *of conducting research and 
develQpment projects OR skills, commitment 



49' 



and flexibility of the project team and their 
participation in project decisions. „ 
% 

3. Use of more part-time project staff , , 

4. Project manager's skills, commitment and 
participation in setting goals.' 

Accidental Factors 

1. Merger, of major channels of development 
often from diverse scientific fields, making 
possible new advances. 

2. / Close working relations of the client organi- 
/ * zations with the project. 



' Appendix C 
Advisory Committee Questionnaire 



The attached five, pages have been provided to facilitate the nom- 
ination process for PROJECT IMPACT . Nominations for case studies will 
be reviewed in order to select .four cases to track for effectiveness or 
impact. 

We have suggested* four categories for' case nomination and provid- 
ed-one page for each. We would like to have your recommendations for 
additions or substitutions in these suggested categories and we have 
provided a fifth page for that purpose. 

^ It might be helpful* for you to review the project description that 
we sent you In August before completing the questionnaire and ask 
yourself, "What projects^ surveys, changes or events have taken place 
in Illinois which would be 'worthwhile case studies in terms of investi- 
gating impact?" 

Please return only the five "Category Pages" to us in the enclosed 
envelope. . ' a 

Thank you, 

PROJECT IMPACT 



. r - 
0 
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PROJECT IMPACT 
Category I: Illinois Curriculum Materials Production 
1 . What was the product? 



2. Where was it produced? 



3. When was it produced? 



4. Who were the key people involved in the production (please give 
names, addresses, and phone numbers if possible)? 



5. Would yoO ' say that this case was an example of (please check 
one): , r , 

, high risk low risk 

6. Would you say that this case was an example' of (please check 
one): 

high cost low cost 

7. Would ^rou say that' this case ,was an example of (please check 
one): " N **^ 

high pay-off where high pay-off was expected 

- hfgrt pay-off where low pay-off was expected . 

low pay-off where high pay-off was expected 

low pay-off where low pay-off was expected ' 

Comments: ' 
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PROJECT IMPACT t 

Category Mr Vocational Education of a New Target Group, in Illinois 
(e.g., handicapped or .incarcerated') * 

I 

1. What was the program, profit, survey or event? 



2. . Where did it occur? 



When did it occur? 



4 



4. Whe were the key people involved in the production (please give 
^ names, addre^es^ and phone numbers if possible)?' 




Would you say that this case was an example of (please check 
one): 

high risk - low risk , 

Would you say that this case was an example of (please check 
one): % * * 

high cost low cost 



7.. Would you say that this case was an example of (please chfeck , 
one): 

high pay-off where high pay-off was expected 

high pay.-off where low pay-off was expected 

low pay-off where high pay-off was, expected 

low pay-off where low pay-off was expected t 



Comments (use reverse side if necessary): 
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PROJECT IMPACT 



Category III: New Concepts of Organization for Vocational Education in 
Illinois (e.g., area vocational schools, ^CETA-YEDPA) 



1. What was the change? > 



2. Where did it occur? 

r ■ 

3. When did it occur? 



4. Who were the key people involved in the production (please giv/e 
names, addresses, and phone numbers if possible)? 



Would you iay that this case was ^an example of (please check 
one): I * 

high risk 1 low risk 

Would you say that this case was an example of (please check 
orte): 

high cost low cost 



7. Would you say that this c^se was an example of (please check 
one): 

high pay-off where high pay-off was expecte^S*__ - 
high pay-off where low pay-off was expected \ 

low pay-off where high pay-off was expected 

low pay-off where low pay-off was expected t 

Comments (use reverse side if necessary):. ^ 
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PROJECT IMPACT 



Category IV: Changes in School Goals, (e.g., career education, ^school- 
operated placement programs) 



1. What was the change? 



2. Where did it occur? 



3. When di'd it occur? 



Who were the key people involved in the production (please give 
names, addresses, and phone, numbers if possible)? 



Would you say that this case was an example of Cplease check 
one): 



high risk 



low risk 



Would you say that this case was an example of (please check 
one): 



high cgst 



low cost 



7. Would you say that* this case was an example of (please check 
one): ' 

high pay-off where high pay-off was expected 

high" pay-off where I6w pay-off was expected 

low pay-off where high pay-off was expected ZZZI 
low pay-off where low pay-off was expected — 



Comments (use reverse side if necessary): 
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Category V: 



PROJECT IMPACT 

(This'^ateg'ory is open for you Jo substitute or add to 
the suggested ones. In your comments below please 
indicate if your recommendation is to add the fifth cate- 
gory or to replace I, 11,21111 or IV,) 



1. What- was the change? 



2. Where did it occur? 



3. When did it occur? 



4. Who were the key- people involved in the production (please give 
* names, 'addresses, and phone numbers if possible)? 



Would you say that this case was an example of*($>lease check 
one): 

high risk ' - low risk 



Wogld you say that this case was an example of v (please check 
one): 

high cost low cost 



Would you say that this case was an example of (please check 
one): 

high pay-off where high pay-off was expected 

higf% pay-off where low pay-off was expected 

low pay-off where high pay-off was expected 

low pay-off where low pay-off was expected \ 



Comments (use reverse side if necessary): 
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Appendix D 

Master Outline for PROJECT IMPACT Case Studies 

PRODUCT DESCRIPTION 
1.1 "Product Characteristics 



1.1.1 Name - 

1.1'. 2 Developer 

1.1.3 Distributor 

1.1.4 Focus 

1.1 .5 Grade Level 

1.1.6 Target Population 
\ 1V1.7 Costs 



1.2 Rationale for Produ c-t 



1.2.1 Long Range Goals of Product 

1.2.2 Objectives of Product 

1.2.3 Philosophy Behind Product 

1.2.4 Theories Supporting Product • . 

1.3 Description of Materials 
— 3 

1.3.1 Organization of Materials 

1.3.2 Format of Materials - (how physically presented) 

1.3.3 Content of Materials (concepts and terminology covered, etc.) 

1.3.4 Cost of Materials to User 

t 1.4 Procedures for Using Product 
1.4.1 Learner Activities 



1.4.1.1 Relationship to program objectives 



* Adapted from drawford et al. (1972) at the American Institute of 
Research . 

( 
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1.4.1.2 Typical activities in a day 
1 .4.1-. 3 Group and individual activities 

1.4.1.4 Kinds of practice, review and feedback * 

1.4.1.5 Recommended period of use 

1.4.1.6 Provisions for motivating student 

1.4.2 Teacher Activities ■ 

1.4.2.1 Teacher strategy 

* IS 

1.4.2.2 Teacher training 

.- » 

^1.4.2.3 Out-of-class preparation 

1.4.3 Provisions for Parent/Community, Involvement 

1.4.3.1 Special Physical Facilities or Equipment 

1.4.3.2 Recommended Assessment Techniques for Users (e,g., 
criterion-referenced, tests, etc.) 

- ./ 2.0 ORIGINS 

* a,. 

x 2.1 . Key Personnel 

2.1.1 Education and Experience of Key Personnel g 

2.1.2 Philosophy of Key Personnel 

2.1.3 Relevant .Research Conducted by Key Personnel 
^ . 2.2 SourcesVof jdeas.for Product 

2.2.1 Trends of the Time ' 

2.2.2 Relevant Research 

2.2.2.1 Theory 

2.2.2.2 Techniques * ' 

2.2.3 Technological Prerequisites 

2.2.4 Similar Products 

2.3 Evolution of Idea for Product 

r 3 r 

A 2.3.1 Formulation of Ideas for Product (Note when, by whorh, why, 
hovy, in what form?) ; , v 

EMC X * 
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2.3.2 Changes in Ideas for Product (Note when, by whom, why, how, 
in what form?) — - ' 

2.3.3 Factors WhLch Stimulated [development of the Idea ' 
5.3.3.1 Efforts of, a key person or persons 

% 2.3.3.2 Available fund ing* v * 

o — 

2.3/3.3 Need for the product (Note how need was defined, 
assessed and documented.) • 

* 2.3.3.4 Potential effectiveness and feasibility of product 

- 2.3.3.5 Motivation to produce product 

- 2.4 Funding for Product 

2.4.1 Initial Efforts to Fund Products 

2.4.2 Contacts with Funding Sources • . 

» 

v(2.4.3 Factors Influencing Funding Sources 
2.C4 Preparation of Proposals 

2.4.5 Details of Funding Agreement 

2.4.6 Description of Funding Sources 

2.4.7 Breakdown of Funds (by stages of development, categories of 
^ use, and/or components). * 
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^.0 PRODUCT DEVELOPMENT 
3.1 Management and Organization 



3.1 .1 Characteristics of Development Agency {Note major funding 
source, age, other projects, nurrJ^r arid qualifications of staff, 
organizational structure)' * ■ * ' 



3.1:2* Relationship of This Product to Agency (Note proportion of re- 
sources, people and facilities devoted to product) 

3.1.3 Other Agencies Involved irVtfevelopment v , ^ 

f * 

3.1.3.1 Characteristics of other involved 'agencies 

3.1.3.2 Relationship to primary developer (Note division of 
responsibility, channels of communication, procedures 
for decision making) 
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' 3.2. Original Development Plan! 

3.2.1 Objectives 

3.2.2 Description of Expected Product 
3.2. 3< Procedures for ProductJDevelopment 

3.2.3, 0.. g8 ,i^ - • . 
3.2.3.2 Tasks 
' 3.2.3.3 Personnel * 
3.2.3-, 4 Time schedule 
3.2.4 Planned Procedures for Product Eva luation 
3.2,4.1 Formative evaluation plans 
3.2^.2 Summativ/er^evaluation plans 

3.2.4.2/1 Modifications of Original • Development Plan 



3.2.5 List and Description of Modifications 

3.2.6 Reasons for Making Modifications 



3^ek)£ 



3.2.7^ Brief Comparison ofj Planned Development with Actual Develop - 



ment 



3.2.\. 1 Actual pt/ocedures for development of product (For each 
.stage/phase of development note the following informa- 
tion) * ' 



4 



3.j?. 8 Development Staff (Note size, education, experience, -special 
qualification^ organizational structure, problems in ^recruiting 
and maintaining. ) 

3*2.9 Patterns of Interaction (Note channels of communication between 
staff r ~general interpersonal relationships.) " r 

3.2.10 Development 

3.2.10.1 Activities and task 



;ks 
/d 



3.2.10.2 Procedures followed (Note effective and not so effective 
•* * ones. ) '. 

.3.2.10.3 Bottlenecks and problems 

3.2.10.4 Major' decisions (Note procedures for making decisions.) 



erJc ( . ' G4 
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3.2.11 Formati ve Evaluation* 
' -? 

3-2.11.1 Conditions of formative evaluation (Note when, by 
whom, with'whom*) 

3.2.11.2 Procedures followed in formative evaluation 

3.2.11.3 Techniques used to gather information 

3.2.11.4 Procedures for modifying product on the basis of forma - 
tive evaluation results 

3.2.11.5 Nature anpl extent &f modifications 

3.2.11.6 Number»and descriptibn of formative evaluation cycles 

3.2.12 Other Formative Evaluation (Note when, by whom, with whom.) 

"7 

3.2.12.1 Conditions of evaluation (Note when, by whom, with 
whom. ) ~ » * 

3.2.12.2 Procedures foll owed 

* ! 5 

3.2.12.3 Techniques used to gather information ✓ 

3.2.12.4 Procedures for modifying product on the basis of evalu - 
ation results 

3:2.12.5 Nature and extent of modifications ^~ 

3.2.12.6 Number and 'description of iterative cycles 

3.2.12.7 Development of Performance Measures/Assessment Tech - 
niques (For e'ath stage/ phase-.-pf development note the* 
following types d^f information.) 

3.2.13 Development Staff (Note size, education, experience,' special 
qualifications, organizational structure, problems in- recKuiting 
and maintaining. ) 

3.2.14 Patterns of Interaction (Note channels of communication between 
staff, general interpersonal relationships.) 

3.2. 15 Development * 

3.2.15.1 Activities and tasks (Note research, on avkviable tech- 
• ' < niques, adaptation of techniques to produdt, expansion 
v of techniques. )• 

3.2.15.2 Procedures followed 1 (Note effective and .not so effective 
ones. ) < 

/ 

.3.2.15.3 Bottlenecks and problems 

I 

i - 



/ 
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3.2.15.4 Major decisions (Note procedures for\naking decisions.) 
3.2.16 Formative Evaluation/^ Assessment Techniques 



* whom . ) 
3.2.16.2 Procedures followed 



ERLC 



3.2.16.3 Techniques used for gathering information ^ 

3.2.16.4 Procedures for modifying assessment techniques • . 

3.2.16.5 Nature and extent of modifications 

3.2.16.6 Number and description of evaluation cycles 

3.2.17 Other Formative Evaluation of Assessment Techniques (e.g., note 
field tests, etc. y - : * 

3.2.17.1 Conditions of evaluation (Note when, by whom, with 
whom. ) i ~~ 

* 

3.2.17.2 Procedures foljowed 

. 3.2.17.3 Techniques used for gathering information • 

3.2.17.4 Procedures for modifying assessment techniques 

r 5 , 

3.2.17.5 Nature and extent of modifications 

■ 7 ? — 

3.2.17.6 Number and description of evaluation cycles 

4.0 SUMMATIVE EVALUATION 
V 4.1 Evaluation Staff t 

4.1.1 Relationship^ to Development Staff 

4.1.2 Size 

^4.1.3 Education, "Experience, Special Qualifications 

4.1.4 Hierarchy ai/d Organizational Structure (Note interpersonal rela- 
tionships. )/ 

4.1.5 Problemy in Recruiting and Maintaining Staff 

4.2 Field Tests 

(For each field test note the following 
types of information) , 

4.2.1 ' Designer of Field Test (Note who and when.) 
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4.2.2 Funding 

4.2.3 . . Coordinator of Field Test (Note who, relationship to developing 
and funding agencies) 

4.2.4 Subjects 

4.2.4.1 Number (students, schools, classes or teachers) 

4.2.4.2 ideographical distribution 

4.2.4.3 Socio-economic description 

4.2.4.4 Selection process (Note schools or districts and experi- 
mental and control ^groups. ) 

*> 

4.2.5 Treatments 

4.2.5.1 Experimental treatments 

r 

4.2.5.2 Contro) treatment 
, , 4.2.6 Measures * * 

4.2.6.1 Description of measures (Note standardized , tests, stSff^ 
constructed^ tests, questionnaires, structured observa* 
tions, school visits,' etc. ) , 1 

4- 2. $.2- Ration^l^ for measures employed 

•*4;2-.6^.3 Procedures for; administration / * 

4.2.7 — Results of Field Test 

4.2.7.1 Analyses used 

4.2.7.2. Rationale for analyses 

* . 4.2.7.3 Student cognitive changes 

4.2.7.4 Student affective changes 

4.2.7.5 Changes in facilitating factors 

4.2.7.6 Unanttcipated .changes 

4.2.7.7 Documentation and reporting of results 
4.2.8 Modifications Made in Product 



\ 



4.2.8.1 Procedures for modifying product on the basis of field 
test results 

4.2.8.2 Natgre and* extent of modifications in product : . 
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4.3 Other Summative Evaluations . 
{e.g. ^ those conducted by the users) 
4.3.-1 Designer of Evaluation Program (Note who and when.) 

4.3.2 Funding 

4.3.3 Coordinator (Note who, relationship to developing and funding 
agencies . ) 

4.3.4 Subjects 

4.3.4.1 Number (students^ schools , classes or teachers) 

4.3.4.2 Geographical distribution • 

4.3.4.3 Socio-economic description 

4.3.4.4 Selection process . (Note schools or districts and experi- 
mental and control groups.) 

4.3.5 Treatments 

: — <s 

v 

4.3.5.1 Experimental treatments 

4.3.5.2 Control treatments % 

4.3.6 Measures 

4.3.6. -1 Description of measures (Note standardized tests, staff 

constructed tests, questionnaires , structured observa- 
tions, school visits, etc.) - . 

4.3.6.2 Rationale for measures employed 

4.3.6.3 Procedures for administration 

4.3.7 Results of 'Evaluation 

4.3.7. -1 Analyses used 

4.3.7.2 Rationale for analyses . * • 

i , ^ . * 

4.3.7.3 Student cogni tive changes 
te — "~< — 

4.3.7.4 Student affective changes 

4.3.7.5 Changes in facilitating factors 

4.3.7.6 Unanticipated t changes 

4.3.7.7 Documentation and reporting of results ^ 
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9 4.3.8 Modifications Made in Product asya Result of Evaluation Results 
4 . 3 . 8* 1 Comments on the adequacy of the evaluations 

5.0 DIFFUSIOI^ 
5.1 Agency Participation 

5.1.1 Agencies Involved (Note characteristics.;) 

5.1.2 Relationships Among Agencies 

5.1.3 Dissusion Activities of Each Agency ' tGive brief descriptions.) 

5.2 Diffusion Strategy 

5.2.1 Developer of Plans/Strategy 

5.2.2 Outline of Strategy - ' 
5.2.2.1' Target 

5.2.2.2 Technigues for reaching target 

5.3 Actual ^Diffusion Efforts 
5!>3.1 Activities (list, describe, when, by whom) 
5.3.2 Responses tb Diffusion /Efforts [ *i 

5.3.2.1 Indications of interest 

5.3.2.2 Early users v * ■% 

5 -3.3 Revisions in Diffusion Strategy (Describe and give rationale for 
making revisions. ) 

5.4 Product Characteristics and Other Factors. Affecting Diffusion 

5.4.1 Complexity of Product . ^ 

5.4;2 Divisibility of Product 

5.4.3^ Compatibility of Product with Other' School Practices 

5.4.4 . Teacher Training Required » 

5-4!5 Ease with Which Product Can be Communicated 

5.4.6 Comparison with Other Products * ~ 

5.4.7 Economic Conditions and Attitudes of tfre Times 
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5.4.8 Cost of Product 

5.4.8.1 Start-up costs 

5.4.8.2 Continuation costs '^ 

5.4.8.3 . Alternative products 

6.0 ADOPTION 

6.1 Extent of Product Use 
(Differentiate between field tests, try outs and adoption) 

6.1.1 Location of Ljsers (Note geographic distribution.) 

6.1.2 Number of Users (Note how many students, schools, number of 
copies sold, etc. ) 

6.1.3 Socio-Economic Characteristics of Users 
6.T.4 Length of Time in Use 

6.1.5 Initiation of Adoption 

6.1.6 Relationship Between User and Developer a nd/or Distributor 

6.2 Installation Procedures 
,6.2.1 Necessary Physical Arrangements or Equipment 

6.2.2 Necessary Classroom Organization * 

' 6.2.3 Importance of Teacher Training 

6.2.3.1 Availability of tnaining^programs 
m 

. a 6-.2.3.2 Development of training programs % 

6.2.3.3 Description o£ training programs, .(Note length, expense, 
organization; materials, techniques used.) 

6.2.3.4 Evaluation of effectiveness of framing 

6.2*4 Extra S taff Requirements . 
V 

6.2.4.1 Supervisory requirements 
. 6.2.4.2 Paraprofessional requirements 
6 . 2 A . 3 Consultants y 

i(l - 



♦ x 
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6.2.5 . Extent of Product Modification Possible 

6.2.6 Degree of Administrative Support Needed 

6.2.7 Importance of Public Relations Effort^ Prior to Adoption 

6.3 Success of Installation Procedures 

6.3.1 Favorable and Unfavorable Conditions r 

6.3.2 Effective and Not So Effective Procedures 

6.3.3 Methods for Obtaining Feedback From Users 

" " i 

7.0 FUTURE OF THE PRODUCT 
7.1 Expected Use or Impact of Product 
7.2 Anticipated Revisions of Product 



>• 
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' Appendix E 

PROJECT IMPACT 
Project. Director Interview Form* 

Project ID#/Title: 

Location/Region : 

Funding Category/ 
Project Type: 

Timelines : 

Funds by FY: 

Number of Partici- 
pants by Group: 

General Information 

1. Describe in one sentence the general purpose of this project. 

2. Where did the idea for this project originate? 

RFP from state department • * LEA Other 

Comment: 

3. Could you list the three major results or products of the project? 

What three factors do you feel were especially important in facili- 
tating the implementation of this project? ( Example: strong sup- 
port from the district superintendent, teamwork among project 
, staff, etc.) 





* Adapted from McCaslin et al. (1976) of, the Kentucky Bureau of Voca- 
tional Education. 
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5. What major barriers or roadblocks were encountered while imple- 
menting this project? (Example: insufficient time to fulfill project 
objectives, unclear project goals, etc.) 



A. PROJECT RESOURCES 



6. Were the support services (secretarial, printing, graphics) suffi- 
cient for implementing the project? 



Yes No 



Comment: 



7. Were the number of paid project staff who worked on the project 
sufficient for implementing the project objectives? 



Yes No 



Comment: 



8. Was the amount of time allocated for implementing the project suf- 
ficient?' 



1_ Yes No 



Comment: 



t 

A 



* Note to interviewer: Throughout the questionnaire when you receive 
' negative answers, ask if they think it affected the outcome of the 
project. 



73 



ERIC 



69 



9. Was the amount of time for responding to the'RFP sufficient? 

[ Yes No 

Comment: • - 



10. Were there sufficient frfids to implement the objectives of the pro- 
ject? f 



Yes No 



Comment : 



^) 11. Were funds provided in a timely manner? 



Yes 



No 



Comment: 



12. 'Was sufficient flexibility allowed by the funding agency in use of 
the funds? 



Yes 



No 



Comment: 
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13. Were equipment, supplies and space sufficient for implementing the 
projeQt? 

Yes No : * 



Comment: 



B. PROJECT STAFF 



14. What was your Job position, title, and area of specializatipn before 
you became the project director? 



t 

15. How many professional* staff members worked on this project? 
Gomment: • 



16. How many of the project staff were: 

Full-time ' Part-time (what %) . 



Comment: 



/ 

\ 



* Excluding clerical help. 
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17. How many of the professional staff members had eithfer previous 
experience on a similar R&D project or preservice training di- 
rectly relating to the project objectives? 



Comments: 



18. In your opinion, was this project characterized by a high level of 
communication : 

a. among project, staff members? : Yes No , NA 

b. with the project monitor? Yes No 'NA 

Comment: 
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19. Was any project related mservice training provided for'the project 
staff during the course of the project? 

Yes * No 



Comment: 



20. Was there any. turnover in professional staff during the' project? 
Yes No 

^ I yes, how many staff members changed* during the course of the 
project? 



\ 



7C 



72 

/ 

Did the project enjoy a high level of visibility and acceptance with- 
in your school district? 

Ves No N A 

1 1 «j 

Comment: ' 



^ C\ ORGANIZATIONAL CLIMAT£^ 



Did you receive backup and moral support for the project, from: 

a. The chief administrative officer for the district? (or agency 
director?) * 

Yes No NA 



Comment: 

/ \ X- 

b. The building principal? (or your immediate supervisor?) 

' Yes No NA 

Comment: 



Have thfe administrators and staff of this school district^ODr agency) 
demonstrated commitment to research and exemplary efforts? 

Yes ' No NA 



Comment: 
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24, Has your district or agency participated in R and D projects prior 
to this one? 

\ Yes, ' No N A 

i 

Comment: 



4^ ■ 

25. To whom did the profect director report? 

Project Director reports directly to Research Development 

Unit 

- - Project Director reports to school or agency administration 

Other 



Comment: 



D. PROJECT PLANNING 



26. Were specific objectives for the project written/out? (If yes, ask 
^to see them) 



Yes No 



27. Are the objectives stated in terms of behavioral changes or specific 
outcome's which will be produced? 

Yes No ^ 



Comment: * y 

t 

' X . 

28. Did the project staff, sponsors and participants view the project 
goals similarly? # * 

k * ^ \ 

Yes No * 

o 7S 
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% 28. (Cont'd.) 



V 



If -no, please explain any dfsagreements and Assess their impact -on 
the project. 



29. Were position descriptions prepared to define the roles and func- 
tions? \ m 

a. of project staff members? Yes No 

b. of project monitor? Yes No 

If yes, ask to see them. ^(Comment on their clarity and specificity 
below.) , 

If no, what effect did this have on the project? 



30! Were the tasks which the project hoped to complete clearlv spelled 
out through a PERT chart, Gantt chart or task breakdown^ 

Y6s No * 



If yes, a$k to see it. (Comment on- its completeness and parity 
'■■below. ) 

If no, whaV effect did this have on the project? 
CommervtTN^ 



31. Do you feel that the scope of work for the project was realisitic? 
Yes No ' 



Comment: 



ft 
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32. Was an advisory committee or 6ther community groups used in 
planning or implementing the project? 

Yes No NA " r 



If yes, could you describe the composition of- the group and how it 
assisted the project? „ 



33. Was an evaluation component built into your original project plan? 
(ask to see) 



Yes No ^ NA 
* 



Comment: 



E. PROJECT CHARACTERISTICS 



34. Did the project require changes in any established pattern in the 
school or classroom? * 

Yes a No . NA 



Comment: 



35. Did the project require a time commitment from participants other 
than regular school hours? ' . 

Yes No . NA 



Comment: 

"V 
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36. 



37. 
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Did the project facilitate individual initiative and flexibility on the 
part of participants? (Did participants incorporate their own ideas 
into the project?) 0 



Yes 



No 



NA 



Comment: 



F. PROJECT IMPLEMENTATION 



Did implementing the project require interacting with onqoinq 
school programs? (as opposed to being self-contained) 



Yes 



No 



NA 



If yes, what type of. interaction? 
Comment: 



38. Did the goals of the project change during project implementation? 



Yes 



No 



NA 



o If yes, in what way? 



39. Did you utilize any consultative assistance 4 , during this project? 
Yes No 



'40. What type? 



41. Who provided it? 




) 
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Was it helpful . Yes. No 



C^Qmmen t : 



i 

t 



Did you have thfe freedom tb make decisions and -necessary changes 
in the project as it progressed? 



Yes No' 



Do you think'that the participants of this project (teachers, coun- 
selors, students, etc.) had a clear conception of what the project 
was alT about? " 

c 

Yes No NA:) 



If no, then why not?. 



Comment: 



Do you believe that the project participants were provided with al 
the help (orientation and Pi^eryice) necessary to successfully par- 
ticipate in the project? 

• ' ' & . MA 

Yes . No * NA 



Commeht: 



Were you able to build and maintain the motivation^ project par- 
ticipants? * 4 



Yes 



No NA 



Comment: 
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*■ • G. PROJECT EVALUATION 

r 

47. Did th^" project have ongoing evaluation? 

• Yes " No- NA „ 

* If yes, briefly describe: ' 1 



i * 



r 



, 48., Was the evaluation helpful^ 



Yes, 



No 



NA 



0 o 0 " 



If yes,, briefly describe the changes made as a result. of the eval- 
uation. 



49. Was a fi/ial evaluation, conducted to evaluai 
the project? ^ ^ * ' 



the effectiveness of 



Yes 



No 



NA 




lf/y*es, briefly describe. 



50.* W9uld you describe thef data ^collected in ' the fjnal evaluation as 
being 'hard, objective data (empirical) or soft, subjective data 
(judgmental)? ' <■ . ' * 



Hafd^ objective 



Soft/* subjective 



NA 



51. Do you - believe that the end products (materials, t procedures, 
. * ideas, ^etc!) justified the resources expended? 



Yes 



No 



If no ? the# why? 



■S3 
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i 

52. Do you expect the prdducts of the project to withstand the test of 1 
time in your district? (WiU they still be used in two years?) ^ 

Yes No N A 



53. Did the project meet all its objectives? 

i 

Yes e ' ' No NA 

Comment: * - < 



54. Have positive changes occurred in the knowledge, attitudes, or 
behaviors of participants as a result of the project? 

Yes No NA 

■ — - — j 

If yes, what changes? f_ 



55. Were any activities or incentives built into the project to encourage 
use of the product or outcome beyond the current project? 

Yes/^ No ' NA 



Comment: 



H. RELATIONSHIP WITH THE RDU , 



56. Do you feel that most of the vocational educators in Illinois are 
aware of the fc RDU and its activities? How could awafeness be 

improved? ^ 

Yes No ■ ... 

Comment: *\ . \ 

IN . N 
) 
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What services and assistance provided by the. RDU have been 
help'ful to you? Your, school? 



How could the RDU improve its process for soliciting, reviewing, 
selecting and monitoring R&D projects? 



What additional services might the' RDU perform to facilitate re- 
search and'development in Illinois? 



/ 



x 
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Apperjdix F 
PROJEOt IMPACT /\ 

f 

( 

Participant Questionnaire 

\ 

We hope that you might take time from your busy schedule to 
assist us in this effort. 



Alt items on this questionnaire refer to ine following project: 



•1. Which category of individuals do yo\( belong to in relation to this 
project?, (Please check one.) 

Student Guidance £o\jnselV 



. Teacher f Other 

* (please specify) * 

2. Ape'you currently involved wjth this project? 

Yes > No • 

3. Are you currently using products -(materials, processes' or find- 
ings) developed in this project? 

Yes No* products were developed - 



l 4./ In your opinion, was tjiis project successful^ 

i 

Yes No 



*23 



Why or why not? 



5\ - Was the purpose pf the project cfear to you? 
Yes -No 



6. Do you .believe that you were provided with a& the necessary Jielp 
in fully understanding your part in the project? ■ ^ 



Yes No 



Adapted from McCaslin et al. (1976) qf^the Kentucky Bureau of Voca* i ( 

tional Education 
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Were, you provided with all the physical resources (space, materi- 
als, equipment) to successfully, participate in the project? 

Yes No 



8. Do you feel that ■ the .motivation of the project participants was 
• built and maintained throughout the project? • 

Yes No 



9. , List three (3) things that you thought were strengths of the pro- 
ject. • . <f 

1. • ' < 



2. 



3. 



\ 



10. -List three things that you thought were wetnesses of the project. 



1. 



; .2. 



/ * 



\ 



»3\ 



11. Did the project require changes in any established patterns of the 
school or classroom 7 . , , 

Y-Hl - No • 



12. .Did your project require an excessive amount of your time? 

- u , .« (J 

Yes No ' * 



V 
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13. Do you -feel that ithe project, personally benefited you? 

* ♦ 

Yes No 

' I f yes, in what way? 



N 1 



< ■ 



♦ 



o 



0 



